Test smells 20 years later: detectability, validity, and reliability

Abstract Test smells aim to capture design issues in test code that reduces its maintainability. These have been extensively studied and generally found quite prevalent both human-written automatically generated test-cases. However, most evidence of prevalence is based on specific static detection rules. Although those are the original, conceptual definitions various smells, recent empirical studies indicate developers perceive warnings raised by tools as overly strict non-representative maintainability quality suites. This leads us re-assess smell tools’ accuracy investigate detectability more broadly. Specifically, we construct a hand-annotated dataset spanning hundreds suites written two generation ( EvoSuite JTExpert ) performed multi-stage, cross-validated manual analysis identify presence six types these. We then use this labeling benchmark performance external validity tools—one widely used prior work one recently introduced with express goal match developer perceptions smells. Our results primarily show current vocabulary highly mismatched real concerns: multiple were ubiquitous developer-written tests but virtually never correlated semantic or flaws; machine-generated actually often scored better, reality, suffered from host problems not well-captured Current strategies poorly characterized these suites; particular, older tool’s misclassified over 70% missing instances (false negatives) marking many smell-free smelly positives). common patterns can be improve tools, refine update definition certain highlight yet uncharacterized issues. findings suggest need for (i) appropriate metrics development practice, (ii) accurate evaluated industrial contexts.